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ABSTRACT: Learning analytics (LA) are summaries, visualizations, and analyses of student data 
that could improve learning in multiple ways, for example by supporting teachers. However, not 
much research is available yet concerning how LA may support teachers to diagnose student 
progress and to intervene during student learning activities. There is evidence for two 
mechanisms, namely that LA tools can 1) aggregate information to a manageable level and 
thereby lower information load, and 2) increase the teacher's confidence and specificity 
concerning the diagnosis of the situation. By means of micro-analytic study of teachers' real-time 
use of LA tools, these mechanisms are further unpacked to answer the question how the 
mechanisms occur and how these results influence the broader field of LA. 

Keywords: Learning analytics, CSCL, teacher, regulation 

Editor's Note: As part of the Special Section on Learning Analytics & Learning Theory this article is followed by a 
short commentary on pp. 163-168 that discusses the challenges it faced and successes it achieved in drawing on 
and contributing to theory use in learning analytics. 

1 INTRODUCTION 

Learning analytics (LA) have been defined as "the measurement, collection, analysis and reporting of 
data about learners and their contexts, for purposes of understanding and optimizing learning and the 
environments in which it occurs" (Siemens & Gasevic, 2012). The result of the collection and analysis of 
data should be actionable knowledge; i.e., knowledge that can, for example, be used for decision 
making. LA can thus serve a multitude of functions, such as monitoring, assessment, and 
recommendation (Chatti, Dyckhoff, Schroeder, & Thus, 2012). This broad categorization of LA makes 
clear that LA is not truly a new research field. Many previous studies have aimed at collecting data about 
learners in order to optimize learning, albeit under a different name than LA. For example, identifying 
navigational patterns of how students use a learning environment was used to design meta-navigational 
support to help students navigate and learn science content (Puntambekar & Stylianou, 2005). 

A large part of research concerning LA (and its related fields) has focused on directly supporting 
students. Recently the opportunity of using LA to support teachers has received considerable attention. 
Collection and analysis of learner data can deliver actionable knowledge for teachers on multiple levels. 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


138 


JOURNAL OF LEARNING ANALYTICS 


S °'iLAR 

SOCIETY for LEARNING 
ANALYTICS RESEARCH 

(2015). Learning analytics to support teachers during synchronous CSCL: Balancing between overview and overload. Journal of Learning 
Analytics, 2(2), 138-162. http://dx.doi.org/10.18608/jla.2015.22.ll 

On a macro level, LA can detect patterns from completed courses to warn teachers if students seem to 
be in danger of failing a task or a course (for example the Signals project, see Tanes, Arnold, King, & 
Remnet, 2011, or the Open Academic Analytics Initiative, see Jayaprakash, Moody, Laurfa, Regan, & 
Baron, 2014). On a micro level, LA can be used for real-time assessment to support teachers' moment- 
to-moment decision making while teaching. In short, analytics could "track and record previously 
ephemeral process data, which could benefit assessment for learning in significant new ways" (Knight, 
Buckingham Shum, & Littleton, 2014). 

While many articles describe the technical underpinnings of LA tools, not many empirical studies have 
been conducted yet to study whether and especially how LA can support teachers while regulating 
students' learning processes (Chatti et al., 2012). The aim of the present article is to contribute to this 
knowledge base by reporting on teacher use of different types of LA tools in the context of computer- 
supported collaborative learning (CSCL). The study shows a case of investigating how teachers actually 
use the tools provided to them. 

In the field of CSCL, teacher presence and teacher support currently receive heightened attention 
(Dyckhoff, Zielke, Bultmann, Chatti, & Schroeder, 2012). There is increasing evidence that teacher 
regulation of student learning activities can lead to improved group collaboration. While digital learning 
environments provide tools for students to engage in cognitive and social activities, this does not 
guarantee that students will adequately finish their task nor that they will have high quality discussions 
(Kirschner & Erkens, 2013). If cognitive or social problems are not addressed and resolved in time, the 
collaborative process is hindered. Research shows that adaptivity of teacher regulation, in terms of well- 
timed and well-chosen teacher interventions, can increase the effectivity of collaboration (Coll, Rochera, 
& de Gispert, 2014; Schwarz & Asterhan, 2011). Therefore, if LA can support them during CSCL, teachers 
may be able to support students more effectively. For example, during computer-supported 
collaborative assignments, because group collaboration is logged, LA could provide the teacher with up- 
to-date reports about collaborative processes that would otherwise be hard to keep track of. 

In section 2, I discuss the theoretical background of the study: first, the role of the teacher during CSCL; 
second, the mechanisms that could explain how LA may benefit teachers during CSCL and what evidence 
exists of these mechanisms. This leads to the central goals of the study in terms of trying to understand 
how these mechanisms may be observed in teacher use of LA. 

2 TEACHER REGULATION OF CSCL AND THE ROLE OF LEARNING 
ANALYTICS 

2.1 Teacher Regulation of Student Collaboration 

During CSCL, the role of the teacher is to regulate student activities by offering assistance when needed 
(Johnson & Johnson, 2008; Kaendler, Wiedmann, Rummel, & Spada, 2014). I define teacher regulation 
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as encompassing the teacher's tasks in fostering student collaboration, including teacher diagnosis of 
student activities and providing support when needed (Kaendler et al., 2014). Teachers can play an 
important role during the problem solving activities of groups of students especially when they adapt 
their support to the understanding of the students. The teacher's challenge is to provide just-in-time 
interventions, aligned to the needs of each specific group (Van de Pol, Volman, & Beishuizen, 2010). To 
decide in which groups, how and when to intervene requires that teachers first notice and diagnose the 
progress and quality of the group's activities (Kaendler et al., 2014; Persico, Pozzi, & Sarti, 2010). An 
intervention without diagnosis of the situation within a group can have detrimental effects. For 
example, a study by Chiu (2004) showed that when the teacher was not aware that students already 
understood the task, teacher interventions tended to harm students' subsequent problem solving 
instead of supporting it. Thus the teacher continuously observes and diagnoses student activities, 
leading to interventions when needed (Kaendler et al., 2014). Ideally, diagnosing precedes teacher 
intervention and helps the teacher to decide on the appropriate action in a given situation, at the 
appropriate time. In other words, teachers engage in dynamic decision making, which is an interplay 
between the information offered by the environment (in this case, the digital learning environment and 
its interface), and the way the teacher processes this information (Feldon, 2007, also see Endsley's 
model of situation awareness, Endsley, Bolte, & Jones, 2003). 

The student activities that the teacher can observe within the learning environment form the input for 
the teacher. During synchronous CSCL, having access to student activities in real-time is an advantage 
for the teacher, because the students' collaborative process is no longer a black box. Ideally, the CSCL 
setting makes it possible for the teacher to be constantly aware of the activities in which students are 
engaged, thereby ensuring that the teacher can adapt the given support for each group. However, 
because a typical classroom will include five or six groups of students, and all groups perform multiple 
activities, there is a large amount of information available to the teacher. It can be a challenge for the 
teacher to keep up with all student activities, possibly turning the advantage of an overview of the 
collaboration into an overload of information (Dyckhoff, et al., 2012; Van Leeuwen, Janssen, Erkens, & 
Brekelmans, 2015a). In other words, it is first necessary for the teacher to notice important events, after 
which an interpretation or diagnosis can be formed, possibly leading to an action or intervention (Van Es 
& Sherin, 2002). With the sheer volume of information available, noticing or identifying meaningful 
events is a challenge. A further complicating factor is that some aspects of collaboration are not 
reducible to a single event, instead spanning across time. For example, the amount of effort put in by a 
single group member only becomes apparent as time progresses; it is a cumulative property. Aspects 
such as these therefore require the teacher to monitor the group at multiple time points (Tabak, 2004), 
which again increases the difficulty of adequately diagnosing the group's progress. 

A second challenge associated with teacher regulation of CSCL is that after noticing and diagnosing an 
important event, teachers must decide whether or not to act in terms of intervening in the collaborating 
group. During collaborative learning, there is a strong emphasis on self-regulation of learning activities, 
which means that students for the most part direct and correct their own actions towards completing 
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the task (Stahl, Koschmann, & Suthers, 2006). Sometimes the best way for teachers to act might be not 
to act at all. Teachers must thus decide whether action is truly necessary, and if so, which intervention is 
most appropriate. Having a clear view of student activities and sufficient confidence of the posed 
diagnosis are therefore important. 

2.2 Mechanisms by which LA may Support Teachers during CSCL 

The difficulties described above demonstrate that teachers may benefit from support tools that fulfill a 
function of monitoring and analyzing student activities. LA tools could take this role because they can 
aggregate information to manageable levels and thereby provide teachers with a quick overview of the 
situation. By providing summaries and visualizations of student activities, the information load on the 
working memory decreases. Working memory only has limited capacity (Paas, Tuovinen, Tabbers, & Van 
Gerven, 2003; Sweller, 2010). When less working memory resources are spent on processing incoming 
information, more resources are available for the teacher to spend, for example, on decisions 
concerning interventions and shifting attention between multiple groups. 

The availability of LA could thus mean that teachers can more easily diagnose the group's progress, 
which in turn could lead to increased adaptivity to student needs (Feldon, 2007). Also, aggregating 
aspects of collaboration not reducible to a single event into a visible summary means that information 
about such processes is more easily accessible to the teacher. It is therefore likely that transforming a 
multitude of information into at-a-glance information will decrease the information load imposed on the 
teacher, thereby freeing mental effort that can be spent on helping students. Evidence of this 
mechanism could be that on a general level, teachers have a better overview of the situation and thus 
can target their attention to the groups that need it most. The LA tools, in line with this mechanism, act 
as a sort of indicator or marker, allowing teachers to discover at a glance which groups might need 
additional attention (Schwarz & Asterhan, 2011). Casamayor, Amandi, and Campo (2009) showed that 
teachers detected more collaboration conflicts when they had access to the LA tools. A recent study by 
Chounta and Avouris (2014) suggests that this mechanism is only beneficial to the teacher when the 
teacher experiences high load. If monitoring groups of students does not take up a lot of mental effort, 
for example when the number of groups is low, then the presence of LA tools can be perceived as a 
nuisance. In this case, teachers already have an overview and the LA "mentally compelled [them] to re¬ 
evaluate the automatic evaluation results [by the LA] from their perspective" (p. 18). In other words, 
when there is no need for supporting LA tools, the tools only added to the workload. 

Besides lowering information load, LA tools may also play a role during teacher regulation because they 
may give teachers more confidence about their diagnosis of the situation. The information shown by the 
LA tools could provide additional evidence to enhance the teacher's diagnosis of the situation. Teachers 
continuously monitor student activities and form impressions of the quality and the progress of those 
activities. Sufficient awareness of student activities is needed to adequately adapt to the situation 
(Endsley, Bolte, & Jones, 2003; Schwarz & Asterhan, 2011; Van de Pol et al., 2010). Awareness of 
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information is indeed a commonly encountered goal of LA tools, both for supporting students (Janssen 
& Bodemer, 2013) as well as for supporting teachers (De Laat, Chamrada, & Wegerif, 2008). When there 
is information available that confirms the impression the teacher gets from monitoring student 
activities, or additional information that makes cumulative properties of the collaboration visible, this 
may influence the step from diagnosis to actual intervention in the sense that the diagnosis would 
sooner lead to the decision to offer groups assistance. The other way around, more confidence of a 
diagnosis may also lead the teacher to decide not to intervene, instead letting students work out the 
problem themselves first. Sufficient confidence or awareness, in short, could increase the teacher's 
ability to regulate and scaffold student learning adequately and to regulate activities during 
collaboration. 

Figure 1 schematically visualizes the two mechanisms outlined above. On the left, mechanism 1 
corresponds to the lowering of imposed load when an LA tool aggregates information to a manageable 
level. On the right, mechanism 2 shows that the specific information provided by the LA strengthens 
teacher diagnosis in comparison to a diagnosis that would otherwise be achieved solely by information 
form the interface without LA. 
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Figure 1. Schematic overview of mechanisms by which LA could support 
teachers: aggregating information (1) and enhancing diagnosis (2). 


2.3 Goal of the Present Paper 


Two studies were recently conducted that provide further evidence for the existence of the mechanisms 
discussed in section 2.2. In two experimental studies, teacher regulation in CSCL (in terms of diagnosing 
and intervening) was compared in situations with and without the availability of LA. In the first study, LA 
was used to visualize students' social activities (Van Leeuwen, Janssen, Erkens, & Brekelmans, 2014). 
The results showed that teachers were better able to detect the occurrence of problems when LA was 
available, thereby strengthening the idea that LA gave them an overview of student activities. 
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Additionally, the teachers with access to LA gave more specific explanations of their diagnoses of the 
groups. The interventions that teachers engaged in were relatively more often targeted towards groups 
that experienced problems, and when they did so, they often addressed individual students, possibly 
indicating diagnosis of a specific student's role within the group. The second study involved LA that 
visualized students' cognitive activities (Van Leeuwen, Janssen, Erkens, & Brekelmans, 2015b). In this 
case, no effect of the LA was found on the ability to detect occurring problems. Experienced cognitive 
load was measured with a self-report scale, and no significant differences were found between the 
teachers who had access to LA and those who did not. There were, however, effects on the 
interventions that teachers performed. The mean frequency of interventions was significantly higher 
when teachers had access to LA, and those interventions were relatively more often directed at 
problematic groups than at non-problematic groups. A further effect of the LA was that when teachers 
detected a problematic group, in comparison to the teachers who did not have LA available, they were 
more likely to act upon this detection with an intervention that dealt with the problem the group was 
experiencing. 

Thus, the literature discussed in section 2.2 together with the studies discussed in this section all provide 
evidence of the existence of the supporting role of LA for teaches during CSCL. The goal of the present 
paper is to extend these findings from whether the mechanisms exist to how the mechanisms are 
enacted by teachers. In particular, the focus is on the patterns of teacher usage of LA tools while they 
regulate the activities of collaborating students. By doing so, the aim was to contribute to the 
knowledge base concerning LA, which currently primarily exists of technical reports and is thus 
underrepresented by empirical studies (Chatti et al., 2012). With this study, the aim was to advance our 
understanding of how teachers interact with learning analytics by examining how they regulate CSCL 
and how LA plays a role during this process on a micro scale. The following research question was 
formulated: How do teachers use learning analytics tools during real-time regulation of collaborating 
groups? 

3 METHODOLOGY 

3.1 Design 

The data set on how teachers use learning analytics in real-time was derived from a larger research 
project in which teachers regulated collaborating groups in simulation situations. The simulations were 
conducted using the existing CSCL learning environment called Virtual Collaborative Research Institute 
(VCRI; Jaspers, Broeken, & Erkens, 2004). VCRI includes both an interface for students to solve the task, 
as well as an interface for teachers to monitor all student activity. The simulation version of VCRI re¬ 
played the data collected in classrooms and displayed them to the participating teachers as if the 
student activities were happening in real-time. This setup made it possible to show the same situation 
(vignettes) to all participating teachers. Thus, when a teacher is shown the simulation software, it will 
appear that groups of students are collaborating in real-time, the only difference being that when the 
teacher sends a message to the students, the students will not respond. In the basic condition, the 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


143 


JOURNAL OF LEARNING ANALYTICS 


S °'iLAR 

SOCIETY for LEARNING 
ANALYTICS RESEARCH 

(2015). Learning analytics to support teachers during synchronous CSCL: Balancing between overview and overload. Journal of Learning 
Analytics, 2(2), 138-162. http://dx.doi.org/10.18608/jla.2015.22.ll 

simulation software was essentially the same as VCRI as used in the classroom, i.e., without any 
additional LA tools. Various versions of the simulation software were created by enhancing the learning 
environment with different types of LA tools, which is explained below. Participants were asked to 
regulate student collaboration in various simulation situations, i.e., vignettes, as if it were a real 
classroom setting, by monitoring student activities and intervening when they thought it was necessary. 

To examine the effects of LA tools and to examine how teachers use the LA tools, two studies were 
conducted. In both studies, two versions of the simulation software were created, one of which was the 
basic version of VCRI without any additional LA tools (the control condition). This basic version was 
compared to a version with added LA tools (experimental condition) based on either students' social 
activities or on students' cognitive activities (see Table 1 in section 3.4). To examine the effects of LA, 
teacher regulation in the control condition was compared to that in the experimental condition. The 
gathered data included both quantitative measures such as teachers' judgment of groups at the end of 
each vignette, as well as qualitative measures such as a log file of teachers' real-time actions during the 
simulation. The quantitative measures were used to examine the effects of LA tools on teacher 
regulation of CSCL and the existence of the mechanisms as discussed in section 2.3 (Van Leeuwen et al., 
2014; 2015b). It is the aim of the present study to focus on how the teachers enacted these mechanisms 
by focusing on teachers' real-time use of the LA tools within the generated log files (see section 3.5). 

3.2 Participants 

The data was part of two separate studies with an experimental set-up. In the present study, we focus 
on the experimental conditions, in which participants used LA tools. In the first study, participants had 
access to LA tools concerning students' social activities. The sample consisted of 6 teachers and 8 
student teachers, with a mean age of 45.2 and 23.0 years respectively and mean teaching experience of 

12.2 and 2.4 years respectively. In the second study, participants had access to LA tools concerning 
students' cognitive activities. This sample consisted of 20 educational sciences students who 
participated in the study as a part of the university course in which they were enrolled concerning the 
subject of ICT use in education. These students had background knowledge about education and an 
affinity with educational practice. Their mean age was 19.9 years. 

3.3 Materials; CSCL Environment 

The input data for the simulation vignettes was gathered from collaborating students in secondary 
education. Collaboration was facilitated by the digital learning environment VCRI, in which students 
typically work on ill-defined tasks that require them to discuss task materials, resulting in a 
collaboratively written essay. All materials for solving the task are included within VCRI, i.e., tools for 
communicating and writing as well as the domain specific sources that can be used to solve the task. 
VCRI is used for synchronous collaboration. Figure 2 displays a screenshot of VCRI as it is used by 
students. For example, in history education, one of the assignments was to explore why the Cold War 
had not resulted in a Third World War. The collaborating groups had to read and analyze historical 
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sources (using the Sources-tool), discuss the information (using the Chat-tool), and write a report (using 
the Cowriter-tool). For all group members, these types of activities are all automatically logged by VCRI 
in the form of opening and closing of tools, sent messages, written words in Cowriter, etc. 



Figure 2. Screenshot of VCRI student interface, with Chat-tool (top left). Cowriter- 
tool (down left), Sources-tool (top right), and one opened source (down right) 

A separate interface of VCRI exists for teachers, through which they can monitor group activities. They 
can open each group's Chat-tool and see the content of each group's Cowriter. Through the Chat-tool, 
teachers can join each group's discussion and intervene when they think it is necessary. Multiple screens 
can be opened at once, thus enabling the teacher to monitor multiple groups at once. Collaboration 
between students as well as teacher regulation of student activities thus fully occurred through the 
online learning environment. 

VCRI can be enhanced with several additional LA tools that show information based on the logged 
student activities described above. In the simulation study, participants had access to this teacher 
interface to monitor all student activity, with different types of LA tools. 

3.4 Learning Analytics Tools 

Various types of student learning activities within VCRI were available as input for the LA tools. From 
these activities, both in real-time, continuous properties as well as cumulative properties could be 
derived. For example, in group discussions in the Chat-tool, a continuous property is whether the group 
is on task or not. An example of a cumulative property, on the other hand, is the amount of effort put in 
by each of the individual group members over time. The properties of student activities led to both 
embedded and extracted teacher supporting LA tools (Wise, Zhao, & Hausknecht, 2014). The continuous 
properties were displayed to teachers as LA tools embedded in the existing groups' tools, for example in 
the Chat-tool, so that they would give teachers real-time information on activities in progress. The 
cumulative properties were extracted and displayed as a separate tool within the teacher dashboard in 
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VCRI. The properties displayed by the LA tools were aspects known to be important for learning yet 
which can be problematic for collaborative groups to achieve without regulation (for example, engaging 
in critical yet constructive discussion, Mercer, 2000). A final consideration for the development of the LA 
tools was the layout. Because the goal of the LA tools was to lower the teacher's information load, for all 
LA tools a visual representation was chosen because it is easier and takes less time to understand than, 
for example, a textual representation (Mazza, 2009). 

The first type of LA tools added to VCRI showed information about social aspects of collaboration (see 
Table 1, middle cells). The real-time, embedded tool was the Shared Space, which is a line that, based on 
automatic coding of student messages in the Chat-tool, shows whether there is agreement or 
disagreement within a group by moving from left to right (Janssen, Erkens, & Kanselaar, 2007). The 
extracted tool was the Participation Statistics, which shows the relative contribution of each group 
member to the total activity within the group, based on the number of keystrokes. The second row of 
Table 1 shows screenshots of both tools. 


Table 1. Comparison of teacher interface of learning environment with and without learning analytics. 


Condition 


Embedded LA in Chat-tool 


Extracted LA, available in separate 
teacher dashboard 
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condition: 
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LA type 2: 
Visualizations 
of cognitive 
aspects 


Concept Trail 





Progress Statistics 



The second type of LA tools added to VCRI showed information about cognitive aspects of collaboration. 
Again, there was an embedded and an extracted tool. The Concept Trail, embedded in each group's 
Chat-tool, showed whether and when the groups mentioned task-related concepts in their discussion. 
The Progress Statistics showed how much progress the groups made on the task by displaying the 
number of written words in the Cowriter and in the Chat. Table 1, third row, shows screenshots of these 
two tools. 

These various types of LA tools could be added to or removed from the simulation version of VCRI, 
thereby creating standardized collaborative situations in which LA tools were either available or 
unavailable to the teacher. 

3.5 Analysis of Log Files 

Teachers were asked to regulate student collaboration in various situations, i.e., vignettes. Each vignette 
lasted about eight minutes and included the activities of five collaborating groups. During this time, 
teachers were free to open the available tools that allowed them to monitor the groups, including the 
LA. They could also send messages to each group. Log files were automatically generated for all teacher 
actions during the vignettes, listing time stamps with the corresponding teacher action. Actions logged 
included opening and closing tools, all mouse clicks and scroll actions (in Chat windows), and the 
messages that teachers sent, specifying which group(s) the message concerned. 

The log files were analyzed following the methodology of micro-analytic case studies, which involved 
tracing the activity of each teacher with a non-computational method of log file data analysis (cf. Wise, 
Perera, Hsiao, Speer, & Marbouti, 2012). Step by step, the teacher's actions could be interpreted and 
reconstructed as derived from the log files: what the teacher did and which content the teacher had 
open on the screen. This way, events were sought that illustrated the way in which LA was used by the 
teacher. The Results section describes which indicators were found. 

To illustrate the found patterns of teacher use of the LA tools and their subsequent interventions, the 
list of timestamps and actions were translated into visualizations. The design of the visualizations was 
inspired by the work by Schwarz and Asterhan (2011), which outline the teacher's actions in the context 
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of each collaborating group. In case of the present paper, the five collaborating groups in each vignette 
are used as columns, and a vertical time line indicates the teacher's actions. This way of visualizing 
focuses on the aspect of time and order of events, as well as on the complexity of the teacher's task, 
which is to monitor multiple groups at the same time (Van Leeuwen et al., 2015a). 

4 RESULTS 

For the log file analysis of teacher regulation, the starting point was the two mechanisms explained in 
section 2.2, namely that LA tools can 1) aggregate information to a manageable level and thereby lower 
information load, and 2) enhance the teacher's diagnosis of the situation. How these mechanisms were 
revealed within the log files and how they were actually observed using examples from various 
participants are explained below. 

4.1 Lowering Information Load 

The first proposed mechanism was that LA tools can assist the teacher by lowering the information load 
imposed by the synchronous CSCL learning environment in which the teacher is able to monitor the 
activities of five collaborating groups. Investigating the log files shows at what points during the 
vignettes the teachers used the LA and if the LA tools assisted teachers in maintaining an overview of all 
occurring activities. There were indeed two distinct ways by which the LA tools did fulfill the role of 
summarizing information that helped the teachers while regulating student activities. 

The first finding was that the LA tools could be used by teachers as an easily accessible way to obtain an 
overview of the status within each group in terms of social or cognitive activities. During the vignettes, 
teachers continuously had to make choices about which groups to monitor at any particular time. 
Teachers could focus on one group or on multiple groups at the same time by opening the 
corresponding tool windows. The log files showed that teachers often consulted the extracted LA tools 
that provided statistics about the group and used this information to make a choice about which group 
to zoom in on. Figure 3 shows an example. The teacher in this example (female, age 25) has access to 
the Shared Space and the Participation Statistics, i.e., LA concerning students' social activities. 

At point 1, the teacher requests the content of the text editor for each group, and also consults the 
Statistics for each group within the Coach tool. She takes several minutes to study each group and then 
zooms in on the Chat conversation of group 4 (point 2). Statistics show that one group member is 
contributing less to the task. She reopens the Coach, updates the information about group 4, and 
further examines their discussion. At point 3, she intervenes in the group and prompts the student that 
has been contributing less to get involved with the group's work. At point 4, the teacher again opens the 
information in the Coach tool and moves between group 4 and group 5. At point 5, she updates the 
information about group 5. In this group, one member has been taking the lead in a dominating way. 
The teacher asks him to let the others finish their own part of the assignment. 
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Figure 3. Example participant number 149, LA tools concerning students' social activities. 


The example shows how the teacher uses multiple sources of information to make choices about how to 
regulate the collaborating groups. The LA tools provide information that is part of the input for the 
teacher's diagnosis of the groups. Figure 4 shows another example of the way LA can provide teachers 
with an overview to make comparisons at the class level. 
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Figure 4. Example participant number 363, LA tools concerning students' cognitive activities. 


At point 1, the teacher (female, age 19, LA concerning students' cognitive activities) opens the Coach 
tool and requests statistics for all groups, which shows the teacher the progress groups are making on 
the task. After accessing this information, the teacher opens the Chat tool for group 3, which enables 
her to follow the discussion within that group. After re-opening the Coach tool and revisiting the 
statistics, the teacher sends several messages to group 3. From the Chat conversation, it appeared that 
the students within the group were merely distributing tasks instead of collaborating. Furthermore, the 
statistics showed that task progress was relatively low. The teacher tells the students to collaborate by 
merging their individual work. She also reminds them that they are falling behind the other groups. 
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Finally, with a third intervention she asks the group to stay focused on the task. After this point, the 
teacher continuously switches between information in the Coach tool and the Chat conversation of 
group 5. At 3:35, she gives them a compliment and motivates them to keep up their good work. Lastly, 
the teacher's attention shifts to group 4. The Concept Trail of this group shows that there are some 
concepts that the group has not yet focused on in their discussion. She tells the group that they are 
showing good work and gives them a specific suggestion for their discussion by mentioning one of the 
concepts from the Concept Trail (an example of mechanism 3, see sections below). 

This example shows how the class overview could lead the teacher to groups that might need additional 
attention. Furthermore, the comparison between groups visible at a glance to the teacher can also be 
used in the teacher's communications to the students. By making the group aware of their progress in 
comparison to other groups, student motivation to work on the task may be further increased. Social 
comparison motivates students to set higher standards for themselves (Janssen et a I., 2007; Michinov & 
Primois, 2005). 

4.2 Enhancing Teacher Diagnosis and Subsequent Intervention 

The second proposed mechanism was that information from LA tools may enhance teacher diagnosis of 
the collaborating groups. It was expected that teacher diagnosis of the situation would be more specific 
and that this would make teachers more confident of their diagnosis. In turn, teachers might be better 
able to decide whether intervention within a group is necessary. In the log files, sequences of using the 
LA tools and subsequent interventions were investigated. 

The first finding concerning this mechanism was that the LA tools indeed helped to increase the 
specificity of teacher diagnosis. Figure 5 shows the log file of a teacher (male, age 24) with access to LA 
concerning students' social activities. A typical strategy developed by the participants was first to 
consult the available statistical information to obtain a quick overview of what groups might be 
experiencing problems (as illustrated in section 4.1 as well). As can be seen in Figure 4, at point 1 the 
teacher checks the LA tools for information about participation within the collaborating groups' 
discussion by requesting statistics about the use of the Chat-tool. In this case, the Participation Statistics 
show that in group 4, one of the group members is less active than the other two group members. The 
teacher opens the chat conversations to see what the groups are discussing and whether all group 
members contribute to the conversation (point 2). After monitoring the conversations, the teacher 
sends a message to group 4, in which the collaborative problem occurs (point 3). The teacher specifically 
targets the group member who is contributing less, and kindly asks this particular student to participate. 
The teacher proceeds to request statistics again. He updates the Chat Statistics and also opens the 
statistics for the Cowriter-tool. This further confirms the finding that the student in group 4 is putting in 
less effort than the others. At point 4, the teacher sends several messages. In group 1, he noticed that 
students were showing off-task behaviour, and tells them to focus on task-related matters. Then, he 
encourages group 4 to continue with the assignment and ensures them that he will speak to the student 
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who was contributing less to the task. Possibly prompted by the processes in group 4, the teacher 
advises group 5 to make sure there is an equal distribution of work among all group members. The 
example shows how the information from the LA tools is incorporated into the teacher's diagnosis: 
specific distributions of participation are visualized. 
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Figure 5. Example participant number 160, LA tools concerning students' social activities. 


The second finding was that not only the specificity of teacher diagnosis increased, but that this 
specificity could also be used in the messages (interventions) teachers sent to the groups. In one of the 
studies, for example, teachers had access to the Concept Trail, which visualizes the concepts students 
use in group conversations and thereby indicates the breadth of their discussions (see Baker, 
Andriessen, Lund, Van Amelsvoort & Quignard, 2007). When teachers notice that a discussion is not 
evolving adequately, the Concept Trail may have helped teachers to relate their impression of a 
problematic student discussion to the notion that the breadth of the discussion was lacking. Also, 
because especially the embedded tools (i.e., the Concept Trail and the Shared Space) provide process 
information about student activities, the teacher is more specifically enabled to offer students 
assistance. For example, continuing the example above, based on the Concept Trail the teacher not only 
can tell the students that the breadth of a discussion is narrow, but also give them feedback about 
which specific task-related concepts they have paid less attention to. 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


152 






































JOURNAL OF LEARNING ANALYTICS 


S °^LAR 

SOCIETY for LEARNING 
ANALYTICS RESEARCH 

(2015). Learning analytics to support teachers during synchronous CSCL: Balancing between overview and overload. Journal of Learning 
Analytics, 2(2), 138-162. http://dx.doi.org/10.18608/jla.2015.22.ll 

An example of a teacher with access to the Concept Trail and Progress Statistics is shown in Figure 6. At 
point 1, the teacher (female, age 19) opens the Chat-tool for each group and takes some time to read 
the evolving discussions. At point 2, she pays a compliment to group 2, who are showing a constructive 
discussion and who have mentioned multiple task-related concepts, as displayed by the Concept Trail. 
She shifts to group 1, and scrolls within the Chat tool to look at the history of their conversation. The 
students are collecting arguments about why the Cold War did not evolve into a Third World War. The 
teacher gives them the suggestion that the reasons they have collected so far for the occurrence of war 
may also be seen as counterarguments. In the Concept Trail, it appears that the students have not yet 
discussed the influence of different countries during the Cold War. The teacher reminds them to think of 
this aspect as well at time 3:17. At point 3, the teacher requests Statistics that show each group's 
progress on the task. Group 4 has done relatively little work on writing the essay. The teacher opens 
their Chat conversation and scrolls through their history. It appears that this group has had a rich 
discussion involving many ideas, as is also witnessed by the Concept Trail. The teacher advises them to 
start putting these ideas to paper. 
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Figure 6. Example participant number 365, LA tools concerning students' cognitive activities. 
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Thus, these examples show how LA has enhanced teacher diagnosis as well as subsequent intervention. 
The initial impressions that teachers may have of the situation can be related to the theoretical 
construct made visible by the LA tools, for example the breadth of a discussion. This is in line with 
theories concerning teachers' ability to notice important events. As Sherin and Van Es explain, noticing 
important events involves "making connections between specific classroom interactions and principles 
of learning and teaching that they represent" (2005, p. 477). After diagnosis, the specific information 
from the LA tools was used by the teachers when they gave feedback to the collaborating groups. 

4.3 Regulating Multiple Social Planes 

Section 4.1 already showed how LA allows teachers to process information more easily by providing 
overviews of each group. Based on a comparison of activities at the class level, teachers choose which 
collaborating group to monitor more closely. During interaction with a group, teachers may address 
specific students. Thus, teachers shift their attention between what has been called multiple social 
planes, i.e., the individual student, collaborating groups, and class level (Dillenbourg & Hong, 2008; Looi 
& Song, 2013). The features of the learning environment VCRI (and the simulation version) are designed 
in such a way that this distinction between planes is possible; teachers may monitor one or multiple 
groups at the same time. The same goes for performing interventions: messages can be sent to one or 
multiple groups. Thus, the social planes in this case also include the multi-group level, when teachers 
decide a message is appropriate not for the whole class, but for a subset of collaborating groups. 

Figure 7 shows an example of how LA assists teachers during the regulation of multiple social planes. In 
this case, the teacher (male, age 22) has access to LA concerning students' cognitive activities. The 
teacher first opens the chat windows for all groups and spends some time on following group 
discussions. At point 1, the teacher sends a message to the whole class in which he announces the start 
of the lesson, asks everyone to make clear decisions about collaboration, and to consult the task 
material when there are questions. The teacher zooms in on group 1 but does not intervene. At point 2, 
he consults the statistics to check the progress of each group on the task. The teacher then specifically 
targets groups 1, 3, and 5, who are slightly behind groups 2 and 4. He sends these groups a message in 
which he prompts them to start writing the essay in the text editor. 

After a few minutes, an example can be seen of how the teacher uses the information from the LA in 
interaction with single groups, similar to the mechanism explained in section 4.2. At point 3, the teacher 
uses the Concept Trail to suggest to group 4 that they may discuss the economy and the distribution of 
power in the context of the task. He suggests to group 2 to stay focused on the main question they are 
trying to answer and to think of what ways governing or ruling occurred during the historical time period 
they are studying. The example in Figure 7 as a whole shows how the mechanisms discussed in the 
previous sections tie together in helping teachers to regulate collaborating groups. 
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Figure 7. Example participant number 367, LA tools concerning students' cognitive activities. 

5 DISCUSSION 


5.1 Discussion of Findings 

The goal of LA is to optimize learning by translating learner data into actionable knowledge. In the 
present article, a framework was provided in which student learning is optimized by supporting the 
teacher. The assumption is that supporting teachers to be more adaptive to student needs while they 
formatively assess their progress, will ultimately provide better results in terms of collaboration and 
learning. The specific goal of this paper was to take the existing empirical evidence one step further and 
to move from the question of whether LA supports teachers to how it does so. 
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Teacher regulation was conceptualized as a process in which teachers first notice events, then interpret 
these events by diagnosing the situation, and finally, decide whether action is necessary, and if so, which 
intervention is most appropriate. Challenges for teachers in the context of CSCL were presumed to lie 
especially in the phase of noticing and diagnosing events, and in the transition from diagnosis to 
intervention. The role of LA during teacher regulation was studied in terms of two mechanisms, namely 
that LA tools aggregate information to a manageable level and thereby lower information load, and that 
LA increases the teacher's specificity and confidence in diagnosing the situation. 

This micro-analytic case study helped to uncover these mechanisms and resulted in a description of 
specific ways in which the teacher interacted with LA tools. Considering aggregation of information, the 
LA tools helped the teacher to decide which group(s) to focus on and facilitated making group 
comparisons (section 4.1). This result might indicate that fewer working memory resources were needed 
to filter incoming information from the five collaborating groups. Considering diagnosis, teachers used 
the information from the LA to confirm and extend their diagnoses before intervening within the 
groups. When teachers did decide to act, they used the information from the LA in subsequent 
interventions to increase the specificity of feedback (section 4.2). In a general sense, the conclusion 
could be that LA helped teachers to maintain an overview of the situation. This was further evidenced by 
finding that teachers were able to regulate multiple social planes (section 4.3). 

Interestingly, these mechanisms on a micro level all point to the way that teachers act proactively 
instead of reactively (a difference employed, for example, by Onrubia & Engel, 2012; Vlachopoulos & 
Cowan, 2010). This is an indication of how LA may create space to reflect on the situation and to decide 
more consciously if and how to intervene (Schwarz & Asterhan, 2011). Because simulation vignettes 
were used in this study, students did not respond to teacher interventions and sequences of multiple 
teacher-student utterances were not possible. Therefore, it was not possible to study usage of the LA 
tools in a reactive way (after initiation of a conversation by a student). Further studies in real classrooms 
could examine whether teachers are able to use LA in these situations as well, by using the logged 
history of the collaboration to adapt to the students' requests for support. 

This paper started from the proposition that teachers need to be supported primarily because of the 
high amount of available information they have access to concerning student activities. The presented 
results, in combination with earlier empirical studies, seem to indicate that LA tools can indeed have a 
positive influence on teacher regulation of CSCL through multiple mechanisms. LA can help to preserve 
the balance between overview and overload of information, so that teachers can benefit from process 
information concerning student collaboration. While these results are promising, it is important that 
more research be performed to replicate and validate these findings. Further research is required to 
validate the proposed mechanisms and to address questions such as how these mechanisms interact, 
which teacher characteristics must be taken into account, and what these mechanisms look like in 
authentic classroom situations. 
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5.2 Relevance of Results in a Broader Context 

An important question to end with is to what extent the results are transferable or contribute to the 
research field of LA in a broader context, i.e., contexts other than CSCL. First, this paper focuses on 
teachers instead of students. A rich field of research exists concerning LA aimed at supporting students, 
which shows that LA tools can have a positive influence on, for example, usage of task-related concepts 
and group participation (Janssen & Bodemer, 2013). Increasingly, LA, or so called dashboards, are also 
aimed specifically at teachers (Verbert et al., 2014). This paper reveals ways in which teachers may 
benefit from LA. Some of the reasons why LA tools are beneficial for teachers are applicable to students 
as well. For example, the Concept Trail provides teachers with detailed feedback about student 
discussion. Shown to students, the Concept Trail may, to a large extent, fulfill the same function. A 
general question following from this research is thus which users (in this case, teachers or students) 
benefit most from which tools. The choice between using LA to support students or teachers will 
depend on the intended goal, as students and teachers will most likely benefit from them in different 
ways. For example, LA tools such as Participation Statistics help the teacher to maintain an overview of 
participation rates within each group. Shown to the students, such an overview creates opportunities 
for social comparison, which might increase student motivation (Janssen et al., 2007; Michinov & 
Primois, 2005, see the examples in section 4.1). An interesting direction for future research is to 
investigate the results of using LA tools to support students and teachers simultaneously, compared to a 
situation in which only the teacher has access to these tools. It could be that when LA tools already 
provide some feedback to students, the pressure on teachers to regulate student activities is relieved. 
As Ertmer and Glazewski note, some "scaffolds may serve as intermediate structures that support 
teachers in the task of [...] scaffolding by creating time for reflection before their response is required" 
(2015, p. 100). For example, statistics that show progress on the task could inform groups of their 
progress and help them to regulate their activities if they notice they are falling behind. The teacher can, 
where needed, further support them to resolve their difficulties. Further research is required to point 
out the best way to support students and teachers during CSCL by means of LA. 

Concerning the context of this study, the results were found in a setting in which teachers regulated the 
activities of five small groups. It is worth discussing how the results are of value when the number of 
students is either scaled up or down; for example, when teachers support individual students, or, on the 
other end, when teachers support a large number of students, for example in a massive open online 
course (MOOC). Collaboration is an essential part of CSCL, which in large part shapes the activities of the 
students. When the collaborative element is absent, the social component and the analytics concerning 
this component are no longer relevant. For example, the Participation Statistics and the Shared Space 
are based on processes within groups. In other settings, there may be more need for tools concerning 
cognitive activities. On the other hand, there are reasons to assume that the mechanisms by which the 
teacher is supported in part remain the same. For example, whether teachers regulate an individual or a 
group of students, insight into the development of the student remains important. Thus, LA tools that 
visualize information across time are relevant in multiple contexts. Also, the way in which LA can 
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increase the specificity of teacher interventions is also transferable to other settings. Providing students 
with specific feedback derived from the LA tools is beneficial, because students then receive more 
detailed help on how to improve their learning (Voerman, Meijer, Korthagen, & Simons, 2012). 

In the case of scaling up to a MOOC, it could again be said that there are similarities to the CSCL context. 
Much information is available about the online activities of MOOC participants. Furthermore, social 
planes can be distinguished in MOOCs as well; besides individual participants that the teacher may wish 
to target, there are often particular groups of participants who are similar in their participation 
characteristics (De Boer, Ho, Stump, & Breslow, 2014). Thus, LA tools could play a role here in the sense 
of aggregating information and helping teachers to select in which groups to intervene. In the present 
study, the choice of whether and how to intervene was left entirely to the teacher. The LA tools mainly 
focused on supporting the teacher in the phase of observing and monitoring student activities. The 
sheer number of participants in MOOCs, however, might be a reason to change this aspect. The LA tools 
may alert the teacher when the system detects a situation that needs intervention, and may also 
provide a suggestion for how to intervene. In the Signals project (Tanes et al., 2011), teachers could set 
the rules for these alerts themselves, so that the teacher is still in control rather than the tool. 

6 CONCLUSION 

The aim of this study was to contribute to the knowledge about how teachers use LA tools to regulate 
student activities. The specific goal was to take the existing empirical evidence one step further and to 
move from the question of whether LA tools support teachers to how they do so. Understanding of the 
mechanisms was expanded by making use of a micro-analytic case study, which yielded new insights 
into how LA aggregates information, enhances teacher diagnosis, and helps teachers to regulate 
multiple social planes within a classroom. This study has helped to advance understanding of how 
teachers interact with learning analytics. This is a topic that is not only of relevance to the field of CSCL, 
but also to the wider context of improving educational processes by means of LA. 
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